Image processing device and method, recording medium, and program

ABSTRACT

The present invention relates to an image processing apparatus and method enabling to correct for camera shake given in image capturing, a recording medium, and a program. Discrete data pieces indicating shifts are obtained. The discrete data pieces are interpolated to obtain data indicating a shift every line. By the interpolation, a correction is calculated every line on the basis of the corresponding shift. On the basis of the corrections, pixel data read positions are determined. Pixel data is read based on the determined read positions. When a pixel to be read overlaps two pixels, pixel data of the pixel to be read is produced (interpolated) from pixel data pieces. In this manner, the effects of camera shake are corrected. The present invention is applicable to video cameras.

TECHNICAL FIELD

The present invention relates to an image processing apparatus andmethod, a recording medium, and a program and, more particularly, to animage processing apparatus and method for reducing the effects of camerashake in shooting through a portable imaging apparatus, a recordingmedium, and a program.

BACKGROUND ART

Portable video cameras are coming into common use. Solid-state imagingdevices used in common video cameras include charge-transfer solid-stateimaging devices, typified by a CCD (Charge Coupled Device) sensor, andX-Y addressing solid-state imaging devices, typified by a CMOS(Complementary Metal Oxide Semiconductor) sensor.

The CMOS sensor has lower power consumption than the CCD sensor.Further, the CMOS sensor is driven by a single low voltage supply and iseasily integrated with a peripheral circuit. Accordingly, incorporatingthe CMOS sensor into an image processing apparatus such as a videocamera is under consideration.

It is, however, difficult to record a high-quality moving image or stillimage using the CMOS sensor as an imaging device of an image processingapparatus such as a video camera. As one of the reasons, a capturedimage is distorted by camera shake. For the CCD sensor which has alreadybeen used as an imaging device of an image processing apparatus, asingle value calculated based on camera shake information obtained inone field or one frame is used as a correction required to execute aprocess of reducing the effects of camera shake. Since exposure time forall pixels is uniform, an image is not distorted. Therefore, shiftscaused by camera shake can be corrected using the single value.

On the other hand, the CMOS sensor captures the image of a subject andprocesses the captured image according to the following mechanism. Theimage is distorted by camera shake. The distortion may be caused by thefollowing reason.

In the charge-transfer solid-state imaging device such as the CCDsensor, all pixels are simultaneously exposed, so that pixel data can beread out. In the X-Y addressing solid-state imaging device such as theCMOS sensor, data is read out every pixel or data is sequentially readout every line. In the solid-state imaging device, when data issequentially read out every pixel, the difference in read time betweenpixels in one line is negligibly smaller than that between lines. In thefollowing description, as shown in FIG. 1, the difference in read timebetween lines will be considered.

For example, a CMOS sensor will now be described. The sensor has astructure in which, as shown in FIG. 1, read time for all pixels is 1/60seconds and each frame includes lines 1 to N. Referring to FIG. 2, timet1 is the start of exposing the line 1 and time t2 is the end thereof.Time t3 is the end of exposing the line 2. Time t4 is the end ofexposing the line 3.

The difference between time t2 and time t3 corresponds to timedifference Δt. The difference between time t3 and time t4 alsocorresponds to the time difference Δt. In other words, the timedifference Δt occurs every line with respect to exposure time. In thiscase, therefore, the time difference between the top line 1 and thebottom line N in one frame is approximately 1/60 seconds correspondingto the exposure time. When camera shake is given under condition thatthe exposure time of the bottom line N is delayed from that of the topline 1 in reading one frame, unfortunately, the image of a subject isdistorted due to the differences in exposure time between lines.

The problem in that the captured image of a subject is distorted willnow be described with reference to FIGS. 3 and 4. FIGS. 3 and 4 showexamples of a case where a stationary rectangle is captured as a subjectin the center of a frame. The middle diagram of FIG. 3 shows a normalimage without being distorted, the image being obtained by shooting thesubject through an image processing apparatus without camera shake. Thesubject is captured as a rectangle.

The right diagram of FIG. 3 shows an image distorted by rightwardmovement of the image processing apparatus in shooting the subject.Similarly, the left diagram of FIG. 3 shows an image distorted byleftward movement of the image processing apparatus in shooting thesubject.

The middle diagram of FIG. 4 shows a normal image without beingdistorted, the image being obtained by shooting the subject through theimage processing apparatus without camera shake. The subject is capturedas a rectangle. The upper diagram of FIG. 4 shows an image extendedvertically by upward movement of the image processing apparatus inshooting the subject. The lower diagram of FIG. 4 shows an imagecontracted vertically by downward movement of the image processingapparatus in shooting the subject.

As mentioned above, in the CMOS sensor, exposure time is shifted (i.e.,imaging timing is delayed) every line. If shifts caused by camera shakeare corrected using a single value calculated from camera shakeinformation obtained in one field or one frame, the effects of camerashake cannot completely be eliminated. Unfortunately, an image to begiven to a user may be distorted.

To correct image distortion caused by camera shake, an image processingapparatus for changing a read position every line in order to correctshifts in the horizontal direction and changing the position of a lineto be read in the vertical direction in order to correct shifts in thevertical direction is proposed (refer to, e.g., Japanese UnexaminedPatent Application Publication No. 2001-358999).

According to a method disclosed in Japanese Unexamined PatentApplication Publication No. 2001-358999, it is assumed that camera shakeinformation is obtained every line constituting a frame.Disadvantageously, it is difficult to obtain camera shake informationevery line in terms of the sampling frequency of a sensor for detectingcamera shake.

In addition, according to the method disclosed in Japanese UnexaminedPatent Application Publication No. 2001-358999, unfortunately, shiftscaused by camera shake are corrected in units of pixels.

It is necessary for conventional image processing apparatuses totemporarily store image data of a captured image. Accordingly, theconventional image processing apparatus requires a memory having acapacity capable of storing data of at least one frame. Therefore, thememory capacity cannot be reduced.

DISCLOSURE OF INVENTION

The present invention is made in consideration of the abovecircumstances and it is an object of the present invention to generate acorrection every line on the basis of camera shake information obtainedin one field or one frame to correct shifts caused by camera shake withhigh accuracy. It is another object of the present invention todetermine coefficients for pixel interpolation on the basis of camerashake information to produce pixel data for interpolation, thuscorrecting shifts caused by camera shake with higher accuracy. It isfurther another object to perform camera shake detection for a shorttime to reduce the capacity of a memory to store image data.

The present invention provides an image processing apparatus including:imaging means for capturing an image through an X-Y addressingsolid-state imaging device; output means for detecting given vibrationto generate vibration detection information; converting means forconverting the vibration detection information generated from the outputmeans into shifts indicating the magnitudes of the given vibration;calculating means for calculating corrections from the shifts obtainedby the converting means, the corrections being used to reduce theeffects of the given vibration on the image; interpolating means forinterpolating the shifts or the corrections; and correcting means forcorrecting pixels constituting the image, captured by the imaging means,on the basis of the corrections to output an image composed of thecorrected pixels, wherein the converting means converts the vibrationdetection information into a first shift and a second shift, the firstshift being obtained when a first line is imaged, the second shift beingobtained when a second line is imaged, the interpolating meansinterpolates the first and second shifts to obtain a shift of each linebetween the first and second lines, and the calculating means calculatesa correction every line on the basis of the corresponding shift obtainedby the interpolating means, or wherein the converting means converts thevibration detection information into a first shift and a second shift,the first shift being obtained when a first line is imaged, the secondshift being obtained when a second line is imaged, the calculating meanscalculates a first correction and a second correction from the firstshit and the second shift, respectively, and the interpolating meansinterpolates the first and second corrections to obtain a correction ofeach line between the first and second lines.

On the basis of the corrections, the correcting means may generatecorrected pixels using pixel data of pixels in the image captured by theimaging means, the image corresponding pixels located in correctedpositions.

The converting means may generate each shift on the basis of the middleinformation piece and the previous and subsequent information pieces ofthe vibration detection information pieces generated from the outputmeans for a predetermined time.

The converting means is capable of predicting future vibrationinformation based on past vibration information.

While one image is captured, the calculating means may generate a shiftonce or more times from the converting means, or calculate a correctiononce or more times from the correcting means.

Each of the shift obtained by the converting means and the correctionobtained by the calculating means may include a value corresponding toone pixel or smaller.

The correcting means may interpolate pixels with an accuracy of onepixel or smaller.

The image processing apparatus may further include storage means forstoring only data of the image captured by the imaging means, the databeing required to correct the image in the vertical direction throughthe correcting means.

An area needed to correct shifts of the image may be stored instead ofone image.

The present invention provides an image processing method including: animaging control step of controlling an X-Y addressing solid-stateimaging device to capture an image; a detection step of detecting givenvibration; a conversion step of converting vibration detectioninformation regarding the vibration detected in the detection step intoshifts indicating the magnitudes of the given vibration; a calculationstep of calculating corrections based on the shifts obtained in theconversion step to reduce the effects of the given vibration on theimage; an interpolation step of interpolating the shifts or thecorrections; and a correction step of correcting the image, capturedunder control in the imaging control step, on the basis of thecorrections to generate the corrected image, wherein the vibrationdetection information is converted into a first shift and a second shiftin the conversion step, the first shift being obtained when a first lineis imaged, the second shift being obtained, the first and second shiftsare interpolated to obtain a shift of each line between the first andsecond lines in the interpolation step, and a correction of each line iscalculated in the calculation step on the basis of the correspondingshift obtained in the interpolation step, or wherein the vibrationdetection information is converted into a first shift and a second shiftin the conversion step, the first shift being obtained when a first lineis imaged, the second shift being obtained when a second line is imaged,a first correction and a second correction are calculated from the firstshift and the second shift, respectively, in the calculation step, andthe first and second corrections are interpolated to obtain a correctionof each line between the first and second lines in the interpolationstep.

The present invention provides a recording medium storing a program foran image processing apparatus which corrects an image captured throughan X-Y addressing solid-state imaging device to reduce the effects ofgiven vibration on the image, the program including: an imaging controlstep of controlling the imaging device to capture an image; a detectionstep of detecting given vibration; a conversion step of convertingvibration detection information regarding the vibration detected in thedetection step into shifts indicating the magnitudes of the givenvibration; a calculation step of calculating corrections in units ofpixels on the basis of the shifts obtained in the conversion step, thecorrections being used to reduce the effects of the given vibration onthe image; an interpolation step of interpolating the shifts or thecorrections; and a correction step of correcting the image, capturedunder control in the imaging control step, on the basis of thecorrections to generate the corrected image, wherein the vibrationdetection information is converted into a first shift and a second shiftin the conversion step, the first shift being obtained when a first lineis imaged, the second shift being obtained when a second line is imaged,the first and second shifts are interpolated to obtain a shift of eachline between the first and second lines in the interpolation step, and acorrection of each line is calculated in the calculation step on thebasis of the corresponding shift obtained in the interpolation step, orwherein the vibration detection information is converted into a firstshift and a second shift in the conversion step, the first shift beingobtained when a first line is imaged, the second shift being obtainedwhen a second line is imaged, a first correction and a second correctionare calculated from the first shift and the second shift, respectively,in the calculation step, and the first and second corrections areinterpolated to obtain a correction of each line between the first andsecond lines in the interpolation step.

The present invention provides a program including: an imaging controlstep of controlling an X-Y addressing solid-state imaging device tocapture an image; a detection step of detecting given vibration; aconversion step of converting vibration detection information regardingthe vibration detected in the detection step into shifts indicating themagnitudes of the given vibration; a calculation step of calculatingcorrections in units of pixels on the basis of the shifts obtained inthe conversion step, the corrections being used to reduce the effects ofthe given vibration on the image; an interpolation step of interpolatingthe shifts or the corrections; and a correction step of correcting theimage, captured under control in the imaging control step, on the basisof the corrections to generate the corrected image, wherein thevibration detection information is converted into a first shift and asecond shift in the conversion step, the first shift being obtained whena first line is imaged, the second shift being obtained when a secondline is imaged, the first and second shifts are interpolated to obtain ashift of each line between the first and second lines in theinterpolation step, and a correction of each line is calculated in thecalculation step on the basis of the corresponding shift obtained in theinterpolation step, or wherein the vibration detection information isconverted into a first shift and a second shift in the conversion step,the first shift being obtained when a first line is imaged, the secondshift being obtained when a second line is imaged, a first correctionand a second correction are calculated from the first shift and thesecond shift, respectively, in the calculation step, and the first andsecond corrections are interpolated to obtain a correction of each linebetween the first and second lines in the interpolation step.

According to the present invention, on the basis of information relatingto given vibration, correction needed for correction to cancel out thevibration is obtained every line constituting a frame by calculation andinterpolation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram explaining the readout of image data from an X-Yaddressing imaging device.

FIG. 2 is a diagram explaining the differences in readout between lines.

FIG. 3 includes diagrams explaining the distortion of a subject imagecaused by camera movement.

FIG. 4 includes diagrams explaining the distortion of a subject imagecaused by camera movement.

FIG. 5 is a block diagram of the structure of an image processingapparatus according to an embodiment of the present invention.

FIG. 6 is a diagram explaining shifts.

FIG. 7 is a diagram explaining the distortion of a subject image causedby camera shake.

FIG. 8 is a flowchart explaining the operation of the image processingapparatus.

FIG. 9A is a diagram explaining the interpolation of the shifts.

FIG. 9B is a diagram explaining the interpolation of the shifts.

FIG. 9C is a diagram explaining the interpolation of the shifts.

FIG. 10A is a diagram explaining corrections.

FIG. 10B is a diagram explaining the corrections.

FIG. 10C is a diagram explaining the corrections.

FIG. 11 is a diagram explaining read positions.

FIG. 12 is a diagram explaining readout.

FIG. 13A is a diagram explaining pixel interpolation.

FIG. 13B is a diagram explaining pixel interpolation.

FIG. 14 is a diagram explaining readout.

FIG. 15A is a diagram explaining pixel interpolation.

FIG. 15B is a diagram explaining pixel interpolation.

FIG. 16 is a diagram explaining an image including interpolated pixels.

FIG. 17 is a flowchart explaining another operation of the imageprocessing apparatus.

FIG. 18A is a diagram explaining pixel readout.

FIG. 18B is a diagram explaining pixel readout.

FIG. 18C is a diagram explaining pixel readout.

FIG. 18D is a diagram explaining pixel readout.

FIG. 19 is a timing chart explaining information required to calculate ashift.

FIG. 20 is a diagram explaining information required to calculate ashift.

FIG. 21 is a diagram explaining the prediction of a shift.

FIG. 22 is a diagram explaining a medium.

BEST MODE FOR CARRYING OUT THE INVENTION

An embodiment of the present invention will now be described below withreference to the drawings. FIG. 5 is a block diagram of the structure ofan image processing apparatus according to one embodiment of the presentinvention. An image processing apparatus 10 includes an imaging device11 which is, e.g., an X-Y addressing solid-state imaging device (such asa CMOS image sensor). Data of the image of a subject captured by theimaging device 11 is supplied to an AFE (Analog Front End) 12.

The AFE 12 converts the supplied image data into digital image data andthen supplies the resultant data into a signal processing unit 13. Thesignal processing unit 13 calculates a luminance signal and acolor-difference signal from the supplied image data and then suppliesthe calculated signals to an image interpolating unit 14.

Although the image data supplied to the image interpolating unit 14corresponds to data pieces of the subject image captured by the imagingdevice 11, all of the data pieces of the image captured by the imagingdevice 11 are not supplied to the image interpolating unit 14. Only datapieces read at timing generated by a TG (Timing Generator) 15 aresupplied to the image interpolating unit 14.

The data supplied to the image interpolating unit 14 is temporarilystored in a memory 17 under the control of a memory controller 16. Onthe other hand, data stored in the memory 17 is read according to aninstruction of the memory controller 16 and is then supplied to theimage interpolating unit 14. In the image interpolating unit 14, thesupplied data is subjected to an interpolation process, which will bedescribed in detail below, of correcting shifts caused by camera shake.The resultant data is output from the image interpolating unit 14 to arecording medium (not shown) or a display (not shown) in order to recordor display the data.

As vibration given to the image processing apparatus 10, user camerashake will be described as an example. Even when vibration other thancamera shake is given, the present invention can be applied in thatcase.

The image interpolating unit 14, the TG 15, and the memory controller 16control the respective operations on the basis of corrections calculatedby a correction calculating unit 19, the correction being obtained basedon a shift obtained by a camera-shake detecting unit 18.

The camera-shake detecting unit 18 detects camera shake given to theimage processing apparatus 10 in shooting. A method using a sensor suchas an angular rate sensor or a sensorless camera-shake detecting methodbased on image processing is used for camera shake detection. Forexample, when the camera-shake detecting unit 18 includes an angularrate sensor, the angular rate sensor supplies data indicating an angularrate in the pitching direction and data indicating an angular rate inthe yawing direction to the correction calculating unit 19.

On the basis of the detected shift caused by camera shake, thecorrection calculating unit 19 calculates data indicating a correctionused to correct a shift caused by camera shake. In other words, thecorrection calculating unit 19 calculates a correction indicating thenumber of pixels, by which an image is moved to reduce the effects ofgiven camera shake, every line using the supplied data.

As mentioned above with reference to FIGS. 1 and 2, image data issequentially read every line in a solid-state imaging device such as aCMOS image sensor. Therefore, a correction is calculated every line.

It is necessary to calculate a correction every line. Timing, when dataindicating a shift is obtained, depends on the sampling frequency of thecamera-shake detecting unit 18. Data indicating a shift required tocalculate a correction is not always supplied every line.

It is assumed that the sampling frequency of the camera-shake detectingunit 18 is different from a data reading frequency for each line of animage captured by the imaging device 11. To obtain a correction everyline, it is necessary to interpolate data of the sampling points of thecamera-shake detecting unit 18 by any means to obtain corrections whilecalculation is not performed based on the data indicating a shiftsupplied from the camera-shake detecting unit 18.

It is assumed that the sampling frequency of the camera-shake detectingunit 18 matches the data reading frequency for each line of an imagecaptured by the imaging device 11 but the frequencies are out of phase.To obtain a correction every line, it is necessary to interpolate dataindicating corrections between the sampling points of the shakedetecting unit 18.

The interpolation of data indicating corrections will now be described.As an example, it is assumed that camera shake detection (timing whendata indicating a shift is supplied from the camera-shake detecting unit18 to the correction calculating unit 19) is performed three times while100 lines are read. For convenience of explanation, it is assumed that ashift detected by the camera-shake detecting unit 18 can be obtained ineach of lines 1′, 50′, and 100′ as shown in FIG. 6. The line 1′ servingas a reference line in the explanation is one of lines constituting acaptured image.

Referring to FIG. 6, a shift in reading the line 1′ corresponds topixels P1, that in reading the line 50′ corresponds to pixels P2, andthat in reading the line 100′ corresponds to pixels P3.

A shift is a value obtained by converting information supplied from thecamera-shake detecting unit 18 into a value in units of pixels throughthe correction calculating unit 19. A shift obtained by the correctioncalculating unit 19 includes a fraction of one pixel. In the subsequentprocessing steps, a shift can be corrected by one pixel or smaller.

The shifts P1 to P3 denote differences relative to a reference shiftobtained at predetermined timing. The predetermined timing is, e.g.,just after the start of capturing an image. Referring to FIG. 6,rightward movement occurs between the reference and the line 1′. Afterthat, rightward movement occurs between the lines 1′ and 50′. Then,leftward movement occurs between the lines 50′ and 100′.

For instance, it is assumed that a stationary rectangle is captured as asubject. When camera shake causing shifts in FIG. 6 is given to theimage processing apparatus 10, an image shown in FIG. 7 is captured(provided to the user) unless data is corrected. Shifts caused by camerashake are corrected so that the distorted image is corrected and therectangle is ultimately provided as a rectangular image to the user.

In the present embodiment, an X-Y addressing solid-state imaging devicesuch as a CMOS image sensor is used as an example of the imaging device11 (FIG. 5). Assuming that a charge-transfer solid-state imaging devicesuch as a CCD image sensor is used as the imaging device 11, a singlevalue is used to correct an image distorted by camera shake every linein a frame. A system according to the present invention can be appliedin that case (the same value is used in one frame).

In order to correct a shift every line, it is necessary to acquire acorrection every line. Again referring to FIG. 6, however, only theshifts of the lines 1′, 50′, and 100′ are acquired. The reason whydiscrete data is acquired is that the acquisition depends on thecamera-shake detection timing (sampling frequency) of the camera-shakedetecting unit 18. In this case, shifts of the lines 2′ to 49′ and thoseof the lines 51′ to 99′ are not obtained. It is, therefore, necessary toobtain shifts of the respective lines by interpolation in order toexecute the correction process.

An interpolation process (correction calculation process) of thecorrection calculating unit 19 will now be described with reference to aflowchart of FIG. 8. In step S11, the correction calculating unit 19acquires data (camera-shake information used to calculate shifts)indicating shifts detected by the camera-shake detecting unit 18.

As mentioned above, the correction calculating unit 19 acquires thethree shifts in the lines 1′ to 100′. In step S12, shifts in the lines2′ to 99′ (excluding the line 50′) are calculated on the basis of thethree shifts. The calculation will now be described with reference toFIGS. 9A to 9C. Referring to FIG. 9A, for example, the shift regardingthe line 1′ is used for the lines 2′ to 40′, that regarding the line 50′is used for the lines 41′ to 60′, and that regarding the line 100′ isused for the lines 61′ to 99′. In other words, a method for applying ashift regarding one line to the previous and following lines to performinterpolation can be used.

Referring to FIG. 9B, the shifts can be obtained using a linearfunction. In an example of FIG. 9B, the shifts regarding the lines 2′ to49′ are calculated according to a linear function based on the twoshifts regarding the lines 1′ and 50′. The shifts regarding the lines51′ to 99′ are obtained according to a linear function based on the twoshifts regarding the lines 50′ and 100′.

In addition, referring to FIG. 9C, the shifts can be calculated usinganother function that is not a linear function. An example shown in FIG.9C is fundamentally similar to that of FIG. 9B, except that a functionother than a linear function is used. Generally, camera shake does notoccur in the same direction at the same speed with the same magnitude.Accordingly, if interpolation is performed using a linear function thathas a constant rate of change, an error may occur. To more appropriatelyperform interpolation (correction) with higher accuracy, a functionrepresenting a curve as shown in FIG. 9C is preferably used.

Which of the methods shown in FIGS. 9A to 9C is used to calculatecorrections is determined by the throughput of the correctioncalculating unit 19. When the correction calculating unit 19 includes amicrocomputer, the corrections can be calculated (interpolated) usingany one of the methods of FIGS. 9A to 9C in accordance with thethroughput of the microcomputer.

Specifically, if the throughput of the correction calculating unit 19 islow, the corrections can be calculated (interpolated) according to themethod with a small calculation amount shown in FIG. 9A. When thethroughput of the correction calculating unit 19 is relatively high, thecorrections can be calculated (interpolated) according to the methodwith a relatively large calculation amount shown in FIG. 9C.

As mentioned above, after the shift is obtained every line on the basisof the discrete shifts, corrections are calculated in step S13. Acorrection is a value in units of pixels indicating how corrected thecorresponding shift is to cancel out the shift (reduce the effects ofcamera shake).

For example, as shown in FIG. 6, the shift regarding the line 1′corresponds to the pixels P1. In order to cancel out the shiftcorresponding to the pixels P1, an amount with a different absolutevalue and a different sign, namely, pixels −P1 can be calculated as acorrection.

FIGS. 10A to 10C are graphs showing corrections calculated by thecorrection calculating unit 19. FIG. 10A shows the correctionscalculated when the correction calculating unit 19 interpolates theshifts according to the method described with reference to FIG. 9A. FIG.10B shows the corrections calculated when the correction calculatingunit 19 interpolates the shifts according to the method described withreference to FIG. 9B. FIG. 10C shows the corrections calculated when thecorrection calculating unit 19 interpolates the shifts according to themethod described with reference to FIG. 9C.

As mentioned above, after the corrections are calculated, read positionsare determined in step S14 (FIG. 6). After the read position isdetermined, if two pixels are located in that position, a new pixel isobtained by interpolation in step S15. The determination of the readpositions in step S14 and the pixel interpolation in step S15 will nowbe described. For convenience of explanation, steps S14 and S15 will bedescribed below together. Step S14 and the previous steps are performedby the correction calculating unit 19 (FIG. 5) and step S15 is performedby the image interpolating unit 14.

Referring to FIG. 11, the left part shows a graph of the corrections inthe horizontal direction calculated by the correction calculating unit19. The right part shows an example of an image A obtained when camerashake given upon capturing a rectangular subject is not corrected. Themiddle part shows an enlarged portion a of the image shown in the rightpart.

The portion a of the image A includes ten lines 20′ to 29′. A pixel b tobe read is located in the line 23′. A correction is calculated everyline. A read start position in each line is determined on the basis ofdata indicating the correction of the corresponding line, namely, howmany pixels are used to shift a read position.

The read start positions are plotted to form a read start line b asshown in the middle part of FIG. 11. The read start line b in the middlepart of FIG. 11 is linear. However, it is not always linear. In somecases, the read start position of a predetermined line may be deviatedfrom those of the other lines. The read start line b is shown forpurposes of illustration. It is not a line formed as a result of theprocess of the correction calculating unit 19 or a line required toexecute the process.

Referring to FIG. 12, when readout is started along the read start lineb, it is found that an image corresponding to the portion a serving as asegment of one side of the rectangle is corrected as a distortion-freeimage shown in the lower part of FIG. 12.

According to the present embodiment, a shift corresponding to one pixelor smaller can be corrected. When attention is paid to any one of thehorizontal direction and the vertical direction, the pixel b to be readmay be located in one pixel or two pixels. First, the case whereattention is paid to the horizontal direction will be described. FIG.13A shows an example in which the pixel b is located in one pixel. Inthis case of FIG. 13A, pixel data of a pixel G3 is read out as pixeldata of the pixel b.

FIG. 13B shows an example in which the pixel b is located in two or morepixels. Referring to FIG. 13B, the pixel b is located over pixels G2 andG3. In this case, data of the pixel b is produced based on adjacentpixels and the resultant data is read as pixel data of the pixel b.

The case where data of the pixel b is produced based on adjacent pixelswill now be described with reference to FIG. 13B. When the pixel b islocated over the pixels G2 and G3, pixel data of the pixel b is producedusing pixel data of the pixel G2 and that of the pixel G3. In this case,the pixel data of the pixel G2 is multiplied by the proportion of thetotal area of the pixel G2 to the area overlapped with the pixel b andthe pixel data of the pixel G3 is multiplied by the proportion of thetotal area of the pixel G3 to the area overlapped with the pixel b, thusproducing the pixel data of the pixel b.

For example, it is assumed that the pixel data of the pixel G2 is dataD2, that of the pixel G3 is data D3, and the proportion of the totalarea of the pixel G2 to the area overlapped with the pixel b is Kb(0<Kb<1). The pixel data of the pixel b is calculated by the followingexpression. Pixel data of pixel b=D2×Kb+D3×(1−Kb).

As mentioned above, the pixel data of the pixel b to be read can beproduced by multiplying data of each of the pixels, which the pixel boverlaps, by the proportion of overlapping. Alternatively, the pixeldata of the pixel b can be generated on the basis of data of pixels,which the pixel b does not overlap, in addition to the data of theoverlapped pixels.

Referring to FIG. 13B, for example, the pixel data of the pixel b isproduced using pixels G1 to G4. In this case, pixel data pieces (D1 toD4) of the pixels G1 to G4 are multiplied by respective coefficients (K1to K4) uniquely obtained from the proportion of the pixel b overlappingthe pixel G2. The resultant values are added. In other words,calculation is based on the following expression.Data of pixel b=D1×K1+D2×K2+D3×K3+D4×K4.

The image interpolating unit 14 or the correction calculating unit 19stores the coefficients K1 to K4 in a table related to the proportion ofthe pixel b overlapping the pixel G2 (i.e., a value corresponding to afraction of one pixel on condition that a correction includes a pixelsegment corresponding to a fraction of one pixel (e.g., 0.5 when thecalculated correction is 1.5 pixels)). The coefficients can be read fromthe table.

As mentioned above, when the pixel data of the pixel b is produced usinga plurality of pixels, shifts caused by camera shake can be correctedwith higher accuracy. If the pixel b is located at one pixel, the pixeldata thereof can be produced on the basis of pixel data pieces ofadjacent pixels.

In that manner, the process of determining a horizontal read positionand the process of interpolating (producing) pixel data to be read asnecessary are executed. A process of determining a vertical readposition and a process of interpolating (producing) data of a pixel tobe read as necessary will now be described. The processes for thevertical direction are fundamentally similar to the above-mentionedprocesses regarding the horizontal direction.

Referring to FIG. 14, the left part includes a graph showing correctionsin the vertical direction calculated by the correction calculating unit19. The right part of FIG. 14 shows an example of an image A obtained bycapturing a rectangular subject when the effects of upward cameramovement are not corrected, the rectangular image being elongated. Themiddle part of FIG. 14 shows a diagram of an enlarged portion a of theimage in the right part.

The portion a of the image A includes seven lines 20′ to 26′. A pixel bto be read is located in the line 23′. In the middle part of FIG. 14,reference numerals are assigned to lines (below, referred to as originallines as appropriate) in which the image should be located uponcapturing the subject without camera shake. For convenience ofexplanation, gaps between the lines are formed. In the example of FIG.14, since the original lines do not coincide with lines of the capturedimage, the image is included in the gaps between the lines.

In the above-mentioned shift correction in the horizontal direction, aread start position in one line is shifted to correct a shift. In theshift correction in the vertical direction, a line itself is selected tocorrect a shift. In other words, a shift in the vertical direction iscorrected in such a manner that data indicating how many pixels are usedto shift a captured line upward or downward is calculated as acorrection to obtain an original image captured without camera shake,and readout is performed based on the correction.

Similar to the shift correction in the horizontal direction, in thevertical correction, the pixel b to be read may be located in one pixel(line) as shown in FIG. 15A or a plurality of pixels (lines) as shown inFIG. 15B. Regarding an approach for generating pixel data of the pixel bwhen the pixel b is located over a plurality of pixels, since the pixeldata can be calculated using coefficients in a manner similar to theshift correction in the horizontal direction described with reference toFIG. 13B, a description of the approach will be omitted.

After the above interpolation (correction) is performed, the lines ofthe captured image match the original lines as shown in the left part ofFIG. 16. Consequently, as shown in the right part of FIG. 16, thecaptured image is corrected such that a portion serving as a segment ofone side of the rectangle is located on the original line. As mentionedabove, even when vertical camera shake is given in capturing an image,the effects of camera shake can be reduced and the resultant image canbe provided to the user.

As mentioned above, according to the present embodiment, the shiftcorrection in the horizontal direction and that in the verticaldirection (steps S14 and S15 of FIG. 8) are performed.

In the above-mentioned present embodiment, the shifts are interpolatedin step S12 and the corrections are calculated in step S13. In otherwords, discrete data pieces obtained by the camera-shake detecting unit18 are interpolated to generate a shift every line. After that, acorrection is calculated every line.

The effects of camera shake can also be corrected according to aflowchart shown in FIG. 17. In step S21, shifts are obtained from thecamera-shake detecting unit 18. In step S22, corrections for the shiftsare calculated by the correction calculating unit 19. In step S23, thecalculated corrections are interpolated to generate a correction everyline.

In other words, the discrete corrections are calculated based ondiscrete data pieces obtained by the camera-shake detecting unit 18. Thecamera-shake corrections are interpolated to obtain a correction everyline. As mentioned above, when the corrections are calculated and thecorrections are then interpolated, the same advantages as those of thecase where the shifts are interpolated and the corrections are thencalculated using the shifts by interpolation can be obtained.

Since step S24 and the following steps are identical to step S14 and thesubsequent steps mentioned above, a description thereof will be omitted.For steps S22 and S23, since fundamental processing, e.g., calculatingthe corrections using a linear function or a non-linear function isidentical to that in the flowchart of FIG. 8 described above, adescription thereof will be omitted.

As mentioned above, the correction calculating unit 19 calculates thecorrections. As shown in FIG. 5, each correction is supplied to theimage interpolating unit 14, the TG 15, and the memory controller 16.Pixel readout based on the correction through the respective componentswill now be described. Referring to FIG. 18A, a read start position R0is set to a predetermined position in a pixel G1 as an example. Thisexample will now be described.

In this case, the TG 15 controls the readout of pixel data supplied fromthe imaging device 11 to the AFE 12. The control of the pixel datareadout through the TG 15 is performed in units of pixels, whose numberis an integer multiple of one pixel. Accordingly, the readout is startedfrom a read start position R1 as shown in FIG. 18B. In this case, theread start position R1 corresponds to the head of the pixel G1.Therefore, pixel data pieces of the pixel G1 and the following pixelsare supplied to the AFE 12 and the subsequent components as shown inFIG. 18C.

In other words, rough shift correction is performed under the control ofthe TG 15.

Subsequently, the image interpolating unit 14 and the memory controller16 correct a shift by a fraction of one pixel. First, pixel data shownin FIG. 18C is sequentially stored into the memory 17 through the imageinterpolating unit 14 under the control of the memory controller 16. Thememory controller 16 reads the pixel data stored in the memory 17 asnecessary and supplies the read data to the image interpolating unit 14.

The readout under the control of the memory controller 16 and a processof the image interpolating unit 14 will be described. In this case, asshown in the upper part of FIG. 18D, pixel data is read from a readstart position R2 located in the middle of the pixel G1. However, datacannot be read from the middle of the pixel. As shown in the lower partof FIG. 18D, therefore, pixel data pieces of pixels G1′, G2′, . . . areproduced (interpolated) instead of readout and are then generated.

The interpolation process, which has already been described above withreference to FIGS. 13 and 15, is performed such that each pixel data ismultiplied by a coefficient or the like. Pixels are generated(interpolated) in the horizontal and vertical directions. According tothe processes, fine shift correction is performed. The memory 17 tostore pixel data is required to the fine correction. The memorycontroller 16 is also required to control the writing and readingto/from the memory 17. Therefore, the present apparatus has thestructure shown in FIG. 5.

In the example in FIG. 5, pixel data (image data) output from the signalprocessing unit 13 is stored in the memory 17 through the imageinterpolating unit 14 under the control of the memory controller 16.According to the structure, before the above-mentioned pixelinterpolation by the image interpolating unit 14 (when data is suppliedfrom the signal processing unit 13 to the memory controller 16), theresolution of an image is converted and the image with the convertedresolution can be subjected to camera shake correction.

In other words, the image interpolating unit 14 can convert theresolution of input data or interpolate pixels of output data. Asmentioned above, when the image interpolating unit 14 can process bothof pixel data (image data) to be stored in the memory 17 and the storedpixel data, process variations can be realized.

In the above-mentioned embodiment, the case where the image processingapparatus 10 includes the TG 15 has been described as an example. The TG15 is not always necessary for shift correction. The present inventioncan be applied in a case where the image processing apparatus 10 doesnot include the TG 15.

When the image processing apparatus 10 does not include the TG 15, theabove-mentioned pixel interpolation (readout and generation) can beperformed by the image interpolating unit 14 and the memory controller16. When a read start position based on the correction calculated by thecorrection calculating unit 19 corresponds to the read start position R0in FIG. 18A, pixel data generation as shown in FIG. 18D can be startedfrom the pixel G1 without using the rough correction by the TG 15. Inother words, both of the rough correction and the fine correction can beperformed by the image interpolating unit 14 and the memory controller16.

In the above-mentioned embodiment, the capacity of the memory 17 has notbeen described. For example, if the memory 17 can store data of oneimage, the memory 17 has enough capacity. As will be described below,even when the memory 17 has a capacity in which data of one image cannotbe stored, so long as a correction is processed, the shift correctionprocess can be performed. In other words, if the capacity of the memory17 is small, the above-mentioned shift correction process can beperformed with high accuracy.

Shift correction in the horizontal direction will now be described. Thedescription will be made on the assumption that the image processingapparatus 10 includes the camera-shake detecting unit 18 (FIG. 5) fordetecting camera shake information eight times in one field or one frameas shown in FIG. 19. When exposure time is set as shown in FIG. 19,camera shake information of a line 1 can be calculated on the basis ofinformation pieces a1 to a6. In this case, it is assumed that the aboveconditions are set.

As mentioned above, according to the present embodiment, corrections tocorrect shifts caused by given camera shake are interpolated(calculated) using data pieces concerning predetermined lines to providea correction every line. Again referring to FIG. 9B, when the shifts orthe corrections are obtained by interpolation, the shifts regarding thelines 2′ to 49′ and the corrections thereof cannot be obtained withoutusing camera shake information pieces regarding the two lines 1′ and50′.

Referring to FIG. 18, however, when the information pieces a1 to a6 areused as camera shake information pieces of the line i (e.g., the line1′), i.e., when camera shake information pieces (six pieces in thiscase), which can be obtained in the exposure time of the correspondingline, are used, the next shift cannot be calculated until the nextinformation a7 is acquired. In this case, the information a7 is obtainedupon reading a line i (e.g., the line 50′).

In the case where the shift is calculated using six camera shakeinformation pieces, therefore, the interpolation process cannot beperformed with respect to lines 2 to (i−1) (e.g., the lines 2′ to 49′)between the lines 1 and i until the data a7 in the line i is obtained.In other words, according to the above-mentioned camera shake detection,when a line in a predetermined position is read, camera shakeinformation of the line is not obtained. Unfortunately, a position toread pixel data cannot be determined by shift correction. Therefore, itis necessary to execute the reading operation (and the interpolationprocess) after image data is temporarily stored in the memory 17 andcorrections are calculated.

According to the present embodiment, however, the shift regarding theline 1 is calculated based on the information pieces a2 to a5. In otherwords, the information pieces at both ends are eliminated from thecamera shake information pieces which can be obtained for the exposuretime and the shift is calculated from the four intermediate informationpieces.

As mentioned above, in the case where the shift is calculated usingintermediate information pieces (the middle information piece and theprevious and following information pieces of time sequence informationpieces), after the exposure time of the line 1 is over, the informationpieces a3 to a6 required to calculate the shift of the line i arealready detected (obtained). Accordingly, the shift regarding the line ican be calculated at time when the exposure time of the line 1 is over.Thus, the shift concerning each of the lines 2 to (i−1) can be obtained(calculated) on the basis of the shift regarding the line 1 and thatregarding the line i at time when the exposure time of the line 1 isover.

Therefore, if image data is not temporarily stored in the memory 17 inorder to calculate the shifts, the shifts and the corrections can becalculated. Thus, even when image data is not stored in the memory 17,shift correction in the horizontal direction can be preformed.

If the shift regarding the line 1 is calculated using the intermediateinformation pieces a3 and a4, i.e., if the shift of the line 1 isobtained using only two intermediate information pieces, the shifts ofthe lines i to j can be calculated at time when the shift of the line 1is obtained (the exposure time of the line 1 is over).

In this case, the shift of the line 1 is generated based on littleinformation. When a shift is corrected every line additionally using theshifts of other lines (e.g., the lines i and j with respect to the line1), i.e., the shifts of a plurality of lines, the shift of the line 1can be calculated with higher accuracy. Thus, the corrections can beobtained with higher accuracy.

Subsequently, shift correction in the vertical direction will bedescribed. Since the vertical shift correction is fundamentally similarto the foregoing horizontal shift correction, the differencetherebetween will now be described. The vertical shift correction willbe explained by again referring to FIG. 19. Similar to the foregoinghorizontal shift correction, a shift regarding a first line (line 1) iscalculated using, e.g., the information pieces a2 to a5. After exposuretime for the first line is over, the information pieces a3 to a6required to calculate the shift of the line i are already detected(obtained).

Accordingly, the image processing apparatus 10 need not include thememory 17 having a capacity for storing image data of one image as inthe case of the foregoing horizontal shift correction. If the memory 17has a small capacity, shift correction to which the present invention isapplied can be performed with high accuracy.

However, the vertical shift correction differs from the horizontal shiftcorrection in that a plurality of lines have to be processed asdescribed above with reference to FIGS. 14 and 15. Therefore, the imageprocessing apparatus 10 has to include the memory 17 having a capacityfor storing pixel (image) data corresponding to pixels (of, e.g., tenlines) required to interpolate shits and pixels.

As mentioned above, although the image processing apparatus 10 needs toinclude the memory 17, the memory 17 has a capacity for storing oneimage or a smaller capacity. The capacity of the memory 17 can bereduced. Therefore, the capacity of the memory 17, excluding thecapacity necessary to store image data for shift correction, can be usedfor another process. Alternatively, the capacity of the memory 17 isreduced, thus reducing the cost of the memory 17.

As mentioned above, the camera-shake detecting unit 18 (FIG. 5)calculates the shifts (corrections) based on information pieces obtainedfor a short time, thus resulting in a reduction in the capacity of thememory 17. In the above-mentioned embodiment, for ease of explanation,eight camera shake information pieces are detected in one field or oneframe. In the application of the present invention, the number of camerashake detection times is not limited. The present invention can beapplied to shift correction with any camera shake detection times.

In image capturing by the imaging apparatus using an X-Y addressingsolid-state imaging device, image information of each pixel is read justafter exposure time is over. When the approach, whereby the capacity ofthe memory 17 can be reduced, is used as mentioned above, the followingproblem may occur: If the exposure time decreases (shutter speedincreases, i.e., the relation of camera shake detection interval>halfexposure time is satisfied), in some cases, the shift at the next timeis not detected at timing when pixel information is read.

The problem will now be described with reference to FIG. 20. In a firstarea, the center of exposure of each line is located between theinformation pieces a4 and a5 regarding camera shake detection. At thattime, the shift has to be interpolated based on the information piecesa4 and a5. When each line in the first area is read, however, theinformation piece a5 regarding camera shake is not detected.Accordingly, shifts cannot be obtained by interpolation in the firstarea.

In a second area, read time for each line is after detection time of theinformation piece a5. Since the center of exposure for each line islocated between the information pieces a4 and a5 regarding camera shakedetection and the information pieces a4 and a5 regarding camera shakeare already detected, a shift can be obtained every line byinterpolation based on the camera shake information pieces as theinformation pieces a4 and a5.

In a third area, the center of exposure for each line is located afterthe detection time of the information piece a5, i.e., between the camerashake information piece a5 and the information piece a6. The shift ofeach line has to be obtained by interpolation on the basis of theinformation pieces a5 and a6. When each line in the third area is read,however, the information piece a6 regarding camera shake is notdetected. Accordingly, shifts cannot be obtained by interpolation in thethird area.

As mentioned above, when the condition of (camera shake detectioninterval)>(half exposure time) is satisfied, some areas where a shiftcannot be obtained every line by interpolation may occur.

When the sampling frequency for camera shake detection is increased, thecamera shake detection interval is reduced, so that the foregoingproblem can be prevented. However, computational complexity andcommunication complexity increase, resulting in an increase in systemload.

As an approach for reducing the system load and solving theabove-mentioned problem, predicting a shift is used. Referring to FIG.21, for example, in order to calculate a shift F1, past shifts P1 and P2are used, a straight line 1 is obtained based on the shifts P1 and P2,and the shift F1 is calculated as one point on the extension of thestraight line 1.

The approach is one example. The number of past camera shake data piecesused for prediction and a predictive algorithm are not limited.

According to the above-mentioned embodiment, in executing the process ofobtaining a shift (correction) in the vicinity of the upper or lower endof a frame by interpolation, calculation for interpolation can beperformed using shift data pieces of the previous and subsequent fieldsor frames. Consequently, a correction to correct a shift can becalculated with higher accuracy. Thus, shifts can be corrected withhigher accuracy.

In the above embodiment, an image to be captured has not been describedin detail. Either of a still image and a moving image can be used as animage to be captured. In other words, the present invention can beapplied to a camera for capturing still images and a camera forcapturing moving images.

As mentioned above, according to the present invention, data to correcta shift can be obtained every line constituting an image can be obtainedby interpolating a little amount of detected data regarding camerashake. In addition, pixel data is interpolated in order to correct ashift of one pixel or smaller. Thus, the distortion of an image can becorrected every line with an accuracy of one pixel or smaller, thedistortion being caused by camera shake given to an image processingapparatus such as a video camera using an X-Y addressing solid-stateimaging device, e.g., a CMOS image sensor.

According to the present invention, further, the capacity of the memory17 for storing image data of a captured image can be reduced. Naturally,the present invention can be applied to a video camera using acharge-transfer solid-state imaging device such as a CCD image sensor inaddition to the X-Y addressing solid-state imaging device such as a CMOSimage sensor.

The above-mentioned process of correcting for camera shake can beexecuted by hardware having respective functions. The process can alsobe executed by software. When a series of processing steps is executedby software, a computer in which a program including the software isinstalled in special hardware is used. Alternatively, the program isinstalled from a recording medium on, e.g., a general-purpose computeron which various programs are installed to execute various functions.

To explain a recording medium, a personal computer using a recordingmedium will now be described in brief. FIG. 22 shows the internalstructure of a general-purpose personal computer. A CPU (CentralProcessing Unit) 201 of the personal computer executes various processesaccording to a program stored in a ROM (Read Only Memory) 202. Data andprograms required to execute the various processes through the CPU 201are appropriately stored in a RAM (Random Access Memory) 203. An inputunit 206 including a keyboard and/or a mouse is connected to aninput-output interface 205, which outputs a signal, supplied to theinput unit 206, to the CPU 201. An output unit 207 including a displayand/or a speaker is connected to the input-output interface 205.

Further, a storage unit 208 including a hard disk and a communicationunit 209 for transmitting and receiving data to/from another devicethrough a network such as the Internet are connected to the input-outputinterface 205. A drive 210 is used to write and read data to/fromrecording media such as a magnetic disk 231, an optical disk 232, amagneto-optical disk 233, and a semiconductor memory 234.

As shown in FIG. 22, the recording medium includes a package medium, inwhich a program is recorded and which is distributed separately from thepersonal computer to provide the program to a user, such as the magneticdisk 231 (including a flexible disk), the optical disk 232 (including aCD-ROM (Compact Disc-Read Only Memory) or a DVD (Digital VersatileDisc)), the magneto-optical disk 233 (including an MD (Mini-Disc)(registered trademark)), or the semiconductor memory 234. Alternatively,the recording medium includes a hard disk, in which a program is stored,built in a computer provided to a user, the hard disk including the ROM202 or the storage unit 208.

In this specification, steps describing the program provided through amedium include not only processes to be executed in the described orderon a time series basis but also processes which are not always executedon a time series basis and are carried out simultaneously orindividually.

INDUSTRIAL APPLICABILITY

According to the present invention, shifts caused by camera shake givenin image capturing can be corrected.

According to the present invention, shifts can be corrected with higheraccuracy.

According to the present invention, a shift of one pixel or smaller canbe corrected.

According to the present invention, the capacity of a memory for storingimage data required to correct shifts caused by camera shake can bereduced.

According to the present invention, even when exposure time is reduced,the capacity of a memory for storing image data required to correctshifts can be reduced.

1. An image processing apparatus comprising: imaging means for capturingan image through an X-Y addressing solid-state imaging device; outputmeans for detecting given vibration to generate vibration detectioninformation; converting means for converting the vibration detectioninformation generated from the output means into shifts indicating themagnitudes of the given vibration; calculating means for calculatingcorrections from the shifts obtained by the converting means, thecorrections being used to reduce the effects of the given vibration onthe image; interpolating means for interpolating the shifts or thecorrections; and correcting means for correcting the image captured bythe imaging means on the basis of the corrections to output thecorrected image, wherein the converting means converts the vibrationdetection information into a first shift and a second shift, the firstshift being obtained when a first line is imaged, the second shift beingobtained when a second line is imaged, the interpolating meansinterpolates the first and second shifts to obtain a shift of each linebetween the first and second lines, and the calculating means calculatesa correction every line on the basis of the corresponding shift obtainedby the interpolating means, or wherein the converting means converts thevibration detection information into a first shift and a second shift,the first shift being obtained when a first line is imaged, the secondshift being obtained when a second line is imaged, the calculating meanscalculates a first correction and a second correction from the firstshift and the second shift, respectively, and the interpolating meansinterpolates the first and second corrections to obtain a correction ofeach line between the first and second lines.
 2. The image processingapparatus according to claim 1, wherein on the basis of the corrections,the correcting means generates corrected pixels using pixel data ofpixels in the image captured by the imaging means, the imagecorresponding to pixels located in corrected positions.
 3. The imageprocessing apparatus according to claim 1, wherein the converting meansgenerates each shift on the basis of the middle information piece andthe previous and subsequent information pieces of the vibrationdetection information pieces generated from the output means for apredetermined time.
 4. The image processing apparatus according to claim1, wherein the converting means is capable of predicting futurevibration information based on past vibration information.
 5. The imageprocessing apparatus according to claim 1, wherein while one image iscaptured, the calculating means calculates a shift once or more times,or calculates a correction once or more times.
 6. The image processingapparatus according to claim 1, wherein each of the shift obtained bythe converting means and the correction obtained by the calculatingmeans includes a value corresponding to one pixel or smaller.
 7. Theimage processing apparatus according to claim 1, wherein the correctingmeans interpolates pixels with an accuracy of one pixel or smaller. 8.The image processing apparatus according to claim 1, further comprising:storage means for storing only data of the image captured by the imagingmeans, the data being needed to correct the image in the verticaldirection through the correcting means.
 9. The image processingapparatus according to claim 1, wherein an area needed to correct shiftsof the image is stored in said storage means.
 10. An image processingmethod for an image processing apparatus for correcting an imagecaptured through an X-Y addressing solid-state imaging device to reducethe effects of given vibration on the image, the method comprising: animaging control step of controlling the imaging device to capture animage; a detection step of detecting given vibration; a conversion stepof converting vibration detection information regarding the vibrationdetected in the detection step into shifts indicating the magnitudes ofthe given vibration; a calculation step of calculating corrections toreduce the effects of the given vibration on the image on the basis ofthe shifts obtained in the conversion step; an interpolation step ofinterpolating the shifts or the corrections; and a correction step ofcorrecting the image, captured under control in the imaging controlstep, on the basis of the corrections to generate the corrected image,wherein in the conversion step, the vibration detection information isconverted into a first shift and a second shift, the first shift beingobtained when a first line is imaged, the second shift being obtainedwhen a second line is imaged, in the interpolation step, the first andsecond shifts are interpolated to obtain a shift of each line betweenthe first and second lines, and in the calculation step, a correction iscalculated every line on the basis of the corresponding shift obtainedin the interpolation step, or wherein in the conversion step, thevibration detection information is converted into a first shift and asecond shift, the first shift being obtained when a first line isimaged, the second shift being obtained when a second line is imaged, inthe calculation step, a first correction and a second correction arecalculated from the first shift and the second shift, respectively, andin the interpolation step, the first and second corrections areinterpolated to obtain a correction of each line between the first andsecond lines.
 11. A recording medium storing a computer-readable programfor an image processing apparatus which corrects an image capturedthrough an X-Y addressing solid-state imaging device to reduce theeffects of given vibration on the image, the program comprising: animaging control step of controlling the imaging device to capture animage; a detection step of detecting given vibration; a conversion stepof converting vibration detection information regarding the vibrationdetected in the detection step into shifts indicating the magnitudes ofthe given vibration; a calculation step of calculating corrections inunits of pixels on the basis of the shifts obtained in the conversionstep, the corrections being used to reduce the effects of the givenvibration on the image; an interpolation step of interpolating theshifts or the corrections; and a correction step of correcting theimage, captured under control in the imaging control step, on the basisof the corrections to generate the corrected image, wherein in theconversion step, the vibration detection information is converted into afirst shift and a second shift, the first shift being obtained when afirst line is imaged, the second shift being obtained when a second lineis imaged, in the interpolation step, the first and second lines areinterpolated to obtain a shift of each line between the first and secondlines, and in the calculation step, a correction is calculated everyline on the basis of the corresponding shift obtained in theinterpolation step, or wherein in the conversion step, the vibrationdetection information is converted into a first shift and a secondshift, the first shift being obtained when a first line is imaged, thesecond shift being obtained when a second line is imaged, in thecalculation step, a first correction and a second correction arecalculated from the first shift and the second shift, respectively, andin the interpolation step, the first and second corrections areinterpolated to obtain a correction of each line between the first andsecond lines.
 12. A program which is executed on a computer to controlan image processing apparatus for correcting an image captured throughan imaging device in order to reduce the effects of given vibration onthe image, the program comprising: an imaging control step ofcontrolling the imaging device to capture an image, the imaging devicecapturing the image every line; a detection step of detecting givenvibration; a conversion step of converting vibration detectioninformation regarding the vibration detected in the detection step intoshifts indicating the magnitudes of the given vibration; a calculationstep of calculating corrections in units of pixels on the basis of theshifts obtained in the conversion step, the corrections being used toreduce the effects of the given vibration on the image; an interpolationstep of interpolating the shifts or the corrections; and a correctionstep of correcting pixels constituting the image, captured under controlin the imaging control step, on the basis of the corrections to generatean image composed of the corrected pixels, wherein in the conversionstep, the vibration detection information is converted into a firstshift and a second shift, the first shift being obtained when a firstline is imaged, the second shift being obtained when a second line isimaged, in the interpolation step, the first and second lines areinterpolated to obtain a shift of each line between the first and secondlines, and in the calculation step, a correction of each line iscalculated on the basis of the corresponding shift obtained in theinterpolation step, or wherein in the conversion step, the vibrationdetection information is converted into a first shift and a secondshift, the first shift being obtained when a first line is imaged, thesecond shift being obtained when a second line is imaged, in thecalculation step, a first correction and a second correction arecalculated from the first shift and the second shift, respectively, andin the interpolation step, the first and second corrections areinterpolated to obtain a correction of each line between the first andsecond lines.